Virtualized connectivity in a cloud services environment

ABSTRACT

A system and method of providing virtualized connectivity in a cloud services environment. A service provider network defines at least a first virtual private network and a second virtual private network for a respective first customer network and a second customer network. The service provider network includes at least one physical connection with a cloud services provider network where the at least one physical connection includes a first private virtual connection between the first virtual private network and the cloud services provider and a second private virtual connection between the second virtual private network and the cloud services provider.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityto U.S. patent application Ser. No. 13/310,589, entitled “VIRTUALIZEDCONNECTIVITY IN A CLOUD SERVICES ENVIRONMENT,” filed Dec. 2, 2011, theentire contents of which are fully incorporate by reference herein forall purposes. application Ser. No. 13/310,589 claims priority under 35U.S.C. §119(e) to U.S. Provisional Application No. 61/419,797, entitled“PRIVATE DELIVEYR OF LOCALIZED CLOUD SERVICES,” filed Dec. 3, 2010, theentire contents of which are fully incorporated by reference herein forall purposes.

TECHNICAL FIELD

Embodiments presently disclosed generally relate to networkcommunications. More specifically, embodiments herein relate to theprivate delivery of virtualized cloud services.

BACKGROUND

Cloud computing generally involves network-based virtualization ofcomputing resources that are shared over a public network such as theInternet. Some example paradigms of cloud computing include Software asa Service (SaaS), Platform as a Service (PaaS), and Infrastructure as aService (laas), whereby computing applications/software, computingplatforms, and computing infrastructures, respectively, are provided asa service to an end user from one or more computing resources that areremotely located across a network (i.e., in the cloud).

For example, a cloud services provider may make a service, platform orinfrastructure from one or more data centers available to one or morecustomers over the Internet. Due to the scalability and flexibility thatpublic networks offer, conventional cloud computing architecturesutilize public networks (e.g., the Internet) as the medium for providingand sharing virtualized computing resources (e.g., SaaS, PaaS, IaaS,etc.). Many customers, however, do not want to use a public network toaccess the cloud services, due in part to the insecurity of publicnetworks, so a private circuit or line is purchased to provide privateaccess to the cloud services. Cloud providers, in turn, are oftenchallenged with provisioning and managing all of the private physicalconnections to its data centers and services requested by customers.

Although more secure than public networks, private networks do notprovide a scalable and efficient virtualization environment forimplementing cloud computing. As such, there is a need for both a secureand scalable networking environment in which to implement a cloudcomputing architecture. It is with these issues in mind, among others,that various aspects of the present disclosure were conceived anddeveloped.

SUMMARY

One aspect of the present disclosure involves a computer-implementedmethod for providing virtualized cloud services across a serviceprovider network, the computer-implemented method involving providingvirtual private network (VPN) access across a service provider networkfor each of a plurality of customers of the service provider network.The method further involves providing private access across the serviceprovider network between at least one of the plurality of customers andat least one of a plurality of cloud service providers associated withthe service provider network, the private access being over a sharedphysical connection between the service provider network and the atleast one of the plurality of cloud providers.

Another aspect of the present disclosure involves a system of providingvirtualized connectivity in a cloud services environment. A serviceprovider network defines at least a first virtual private network and asecond virtual private network for a respective first customer networkand a second customer network. The service provider network includes atleast one physical connection with a cloud services provider networkwhere the at least one physical connection includes a first privatevirtual connection between the first virtual private network and thecloud services provider and a second private virtual connection betweenthe second virtual private network and the cloud services provider.

BRIEF DESCRIPTION OF DRAWINGS

The foregoing and other objects, features and advantages of theinvention will be apparent from the following description of particularembodiments of the invention, as illustrated in the accompanyingdrawings in which like reference characters refer to the same partsthroughout the different views. The drawings are not necessarily toscale, emphasis instead being placed upon illustrating the principles ofthe invention.

FIG. 1 is a system diagram illustrating one possible implementation of acomputing network providing virtualized connectivity in a cloud servicesenvironment.

FIG. 2 is a block diagram illustrating a provisioning portal andassociated method of establishing a virtual connection between a cloudprovider and an enterprise over a private network, the virtualconnection including a physical connection shared between the enterpriseand other enterprises.

FIG. 3 is a logic diagram illustrating one way to implement the systemillustrated in FIG. 1.

FIG. 4 is a system diagram illustrating one way to implement the logicof FIG. 3, the system of FIG. 4 allowing a plurality of enterprisenetwork offices direct access to a cloud service.

FIG. 5 is a system diagram illustrating one way to implement the logicof FIG. 3, the system of FIG. 5 allowing a plurality of enterprisenetwork offices indirect access to a cloud service when a central officeof the customer network provisions such access.

FIG. 6 is a system diagram illustrating one way to implement the logicof FIG. 3 using a layer 2 connection between an enterprise and a cloudprovider over a provider network.

FIG. 7 is a system diagram similar to FIG. 3 and further includingNetwork Address Translation (NAT) services.

FIG. 8 is a logic diagram illustrating one way to implement the systemillustrated in FIG. 7.

FIG. 9 is a system diagram illustrating one particular way to implementa system as set out if FIG. 1 and further including NAT services withvirtual domains.

FIG. 10 is a diagram providing further detail of the system set forth inFIG. 9.

FIG. 11 is diagram illustrating implementation of class of service inthe private network between a cloud provider and an enterprise.

FIG. 12 is a block diagram of a computer system suitable for performingmethods and implementing systems in accordance with an exampleembodiment.

FIG. 13 is a flow chart that shows processing operations performed inaccordance with an example embodiment.

Throughout the drawing figures, like reference numerals will beunderstood to refer to like parts and components.

DETAILED DESCRIPTION

Aspects of the present disclosure involve virtualized connectivity in acloud services environment. More particularly, private enterprisenetworks are extended to a content or a service source, such a clouddata center, by way of a virtualized private handoff that may eliminatethe necessity of a private circuit or line connection while providingthe customer with the level of privacy and security required. Stateddifferently, aspects of the present disclosure involve private networkaddressing between an enterprise or other cloud customer and a cloudprovider over shared network resources so that more than one enterprisemay obtain private secure access to cloud resources over a sharedconnection between the shared network resources and the cloud provider.Moreover, multiple enterprises may access multiple cloud providers andresources privately over shared network resources by way of privatevirtual connections over shared physical connections.

Embodiments disclosed herein provide a scalable networking environmentfor implementing a cloud computing architecture across a secure privatenetwork. For example, a service provider may provide private networkaccess across its network between two or more customer locations (e.g.,via a Virtual Private Network “VPN”, such as Layer 2 or Layer 3Multiprotocol Label Switching “MPLS” VPNs). According to an exampleembodiment, the service provider can further provide private accessbetween the two or more customer locations in addition to providingprivate access (or a pseudo-private, private/public hybrid type access)to one or more cloud service providers. The cloud service providers canbe situated internally in and/or externally to the service providernetwork. Additionally, example embodiments provide a multitenantapproach whereby two or more customers of the service provider (e.g.,each having their own VPN across the network) have private access (or apseudo-private, private/public hybrid type access) to the one or morecloud service providers.

Furthermore, and assuming that two or more cloud service providers areconfigured to provide private and secure cloud services (e.g., SaaS,PaaS, IaaS, and the like) across a service provider network, embodimentsherein provide segregation (or privacy) between the two or more cloudservice providers and between the two or more customers. For example,assume a first cloud service provider CP1 and a second service providerCP2 that provide respective cloud services to a first enterprise E1 anda second enterprise E2 across a service provider network, wherebyenterprise E1 and enterprise E2 have respective VPNs across the serviceprovider network. In such a multitenant/provider-segregated environment,cloud service provider CP1 would be restricted from accessing (i.e.,cannot ‘see’) data/traffic transceived across the service providernetwork between cloud service provider CP2 and enterprise and E1 and E2.Similarly, cloud service provider CP2 would be restricted from accessingdata/traffic transceived across the service provider network betweencloud service provider CP1 enterprise E1 and E2. Furthermore, enterpriseE1 would not have access to data/traffic transceived between enterpriseE2 and cloud service providers CP1 and CP2. Likewise, enterprise E1would not have access to data/traffic transceived between enterprise E1and cloud service providers CP1 and CP2.

Example embodiments further provide scalability by enablingpublic-to-private and private-to-public network address translation(NAT) between cloud service providers and customers across the serviceprovider network. Embodiments can further leverage functionality offirewalls (FWs), Virtual Local Area Networks (VLANs) and 802.1Qinterfaces, Ethernet-over-MPLS, Virtual Routing and Forwarding (“VRF”,for example, importing and exporting VRF tables/route distinguishers“RDs” throughout the service provider network), as well as other knownprotocols to implement various private and secure connectivityconfigurations between a service provider network, cloud serviceproviders, and customers of the service provider network. Further yet,embodiments herein can utilize Layer 2 VPN functionality to facilitateVirtual Machine (VM) movement and migration in the cloud environment.

FIG. 1 is a system or network level diagram illustrating aspects of thepresent disclosure. In this system, two or more enterprises 10 (alsoreferred to as customers) may access services provided by two or morecloud services providers 12 or other service providers. The use of theterm “enterprise” (or customer) refers to one or more computing devices,which are likely but not necessarily part of a network of computingdevices used by an enterprise, that are configured to access and provideaccess to application, computing and data resources internal and/orexternal to an enterprise. The use of term “cloud service provider”refers to one or more computing devices that are configured to provideapplication, computing and data resources over a network to one or moreenterprises ranging from individuals to multinational corporations withgeographically dispersed offices.

In the system set out in FIG. 1 and discussed relative to otherembodiments disclosed herein, the enterprises 10 and the cloud providersare interconnected through virtual private networks (VPNs) 14established on a common network infrastructure 15, which may bepartially or completely segregated from public network components andinfrastructure such as in the public Internet. In this example,enterprise (E1) may access services from cloud provider (CP1) and cloudprovider (CP2) over a first VPN 14A for the first enterprise, whereasenterprise (EN) may access services from cloud provider (CP1) and cloudprovider (CPN) over a second VPN 14B for enterprise EN. With respect toenterprise (E1), communication paths with cloud provider (CP1) and cloudprovider (CP2) are enabled whereas a path between cloud provider (1) andcloud provider (2) is not enabled (or allowed). Similarly, with respectto enterprise (N), communication paths with cloud provider (CP1) andcloud provider (CP2) are enabled whereas a path between cloud provider(CP1) and cloud provider (CPN) is not enabled (or allowed). Further,communication paths between enterprises are also not enabled (orallowed). Thus, the system allows and enables private specificcommunications between enterprises and cloud providers and prohibitscommunications among enterprises or among cloud providers as well accessin other forms to those networks.

The virtual connections between enterprises and cloud providers may beestablished and removed by way of a portal 16 coupled with the providernetwork implementing the VPNs. In various possible implementations, onephysical connection 17 between the network implementing the VPNs and thecloud provider is shared and the system allows for the provisioning of aplurality of virtual connections between enterprises and the cloudprovider over the physical connection. Stated differently, through theuse of VPNs in the provider network and virtualizing a physicalconnection between the provider network and the cloud provider, aplurality of enterprises may privately access cloud services through theprovider network over a shared physical connection with the cloudprovider. While FIG. 1 illustrates single instances of cloud providersCP1, CP2 and CPN, any particularly cloud provider may have one or moredata centers or computing locations and/or devices geographicallyproximate or remote that provide some form of service, and thereforeseveral different shared connections may be provided between theprovider network and the various cloud provider service locations.Similarly, an enterprise may range from a discrete computing device or anetwork of computing devices geographically dispersed, with differentconfiguration for accessing the network 15 to obtain services from thevarious cloud providers 12.

Unlike conventional systems where each enterprise desiring a privateconnection with a cloud provider must have a dedicated physicalconnection with the cloud provider, aspects of the present disclosureallow a plurality of enterprises to share a physical connection (orline) through the establishment of private virtual (logical)communication channels with the cloud provider over the sharedconnection. The portal can control dynamically turning-up andtearing-down virtual connections to the cloud providers. In variousexamples, the portal can manage the importing and exporting of routetargets through the provider network and at the enterprise and cloudprovider routers, manage the provision of VLANs between the cloudproviders and the provider network, and manage the provisioning of NATservices within the provider network and between the cloud provider andthe enterprise. Besides the privacy, security, and flexibility inconnecting with various cloud providers without having to establish aphysical connection, as the network is shared by the various enterprisesand cloud providers is not part of a public network, the system alsoallows communications between an enterprise and a cloud provider toavoid much of the latency associated with conventional cloud computingthat work over a public network such as the Internet, in one example,because the private network can manage traffic flows between thenearest, logically and/or geographically, enterprise device requestingthe service and cloud provider location offering the requested service.

The portal 16, which also may be referred to as a customer managementmodule, performs functionality including, but not limited to,collecting, aggregating, filtering, managing, configuring, reporting,and presenting layer 4+(e.g., transport layer, application layer, etc.)data/statistics from the network to customers and/or cloud serviceproviders. For example, the portal can collect determine Quality ofService (Qos) data (e.g., metrics, statistics, etc.) associated with aparticular customer and then report and/or present the QoS data to theparticular customer. Assuming that the network offers a variety of QoSlevels to customers, the customer management module can enable aparticular customer to select one or more QoS levels (or class ofservice) from the variety of offered QoS levels. The customer managementmodule can also enable a particular enterprise or other customer toselect from among the cloud service providers for establishing a‘connection’ to one or more cloud service providers. The customermanagement module can also enable a particular customer to select one ormore services (e.g., SaaS, laaS, PaaS, etc.) associated with each cloudservice provider such that the particular customer will be able toutilize the each selected service from across the network (i.e., cloud).The customer management module can also provide a customer portal (e.g.,web portal) that enables a particular customer to view and manage thevarious QoS levels, cloud service providers, services, etc., associatedwith the particular customer. The customer management module can alsoprovide various selectable cloud packages that include, for example, oneor more of QoS levels, cloud service providers, services, pricingmodels, downgrades/upgrades, etc., or any combination or associationthereof, from which a customer can select (e.g., via the portal).

The VPN provisioning portal 16, shown in more detail in FIG. 2, alsoacts as a virtual marketplace for enterprises or other consumers ofcloud services to select and obtain access to a cloud provider. When anenterprise or other customer logs in to the portal, a customerapplication 18 is launched that presents the enterprise user with a listof available cloud providers 20 as well as a list of existing cloudproviders 22 (i.e., those cloud providers that the enterprise isaccessing by way of the systems described herein). Some cloud providersmay avail their services to all enterprises, while other cloud providersmay have membership limitations (e.g., a cloud provider may only provideservices to health care enterprises). The portal 16 also provides amechanism for cloud providers to register and make themselves availablefor enterprises using the system, and to monitor and processtransactions with enterprises. For example, as shown in FIG. 2, theportal may also include a cloud provider application 24 that allows thecloud provider to manage its interaction with the system and customers..

To obtain services from a new cloud provider using the system, inoperation, the enterprise requests a connection to a cloud provider(operation 200). The request may be transmitted to a cloud providerapplication, which may reside on the portal 16 or otherwise. Forexample, the portal may include functionality accessible by the cloudprovider allowing the cloud provider to review its accounts, process newcustomers, analyze performance statistics and the like. The cloudprovider application may include an identification of pending requests26 as well as information concerning the requests. The cloud providerprocesses the request and returns a price for the requested service(operation 210). The portal applications 18, 24 are configured to allowthe enterprise and cloud provider to negotiate pricing, bandwidth, term,and other aspects of the relationship between the enterprise and thecloud provider. When the terms are accepted (operation 220), the systemprovisions the new connection between the enterprise and the cloudprovider (operation 230).

FIG. 3 is a system diagram illustrating one possible implementationconforming to aspects of the present application. In this example,similar to FIG. 1, three cloud providers CP1-CP3 are illustrated alongwith three customers or enterprises E1-E3. At the various cloud providerlocations, an extranet hub 28 is shown. An extranet hub may include onEthernet hub, a switch, a router or other piece of equipment where thecloud provider is connected with the private network. In this examplesystem and for purposes of illustrating aspects of the system,enterprise E1 includes connections with cloud providers CP1, CP2, andCP3 over a first VPN 34, enterprise E2 has no connections with a cloudprovider, and enterprise E3 has only one connection with cloud providerCP1 over a second VPN 36. Each of the VPNs may be implemented on aservice provider network 38. As shown, enterprise E1 and enterprise E3share a common physical connection 32 to cloud provider CP1.

Each cloud provider has a unique identifier associated with it. In thepresent example, the unique identifiers 1:1, 1:2, 1:3 and 1:4 areassociated with cloud providers CP1, CP1, CP2 and CP3, respectively.Similarly, each customer, e.g., enterprises E1-E3, also has their ownidentifiers. In the present example, the unique identifiers 1:100,1:200, and 1:300 are associated with enterprises E1, E2, and E3,respectively. Of note, CP1 includes the first identifier 1:1 associatedwith private communications with enterprise E1 and the second identifier1:2 associated with private communications with enterprise E3. Eachenterprise (or customer) shown in FIG. 3 is a logical representation ofthe enterprises computing network, which may be a local area network, awide area network, or other forms of networks and combinations thereof,through which various users of various possible computing devices, suchas personal computers, terminals, thin clients, tablets, smart phones,etc., may access various possible provisioned cloud service. Moreover,the computing network may include geographically dispersed computingelements. Private communications between a given cloud provider and anenterprise may be on restricted to a particular router or device withthe customer network or may be available to a plurality of routers ordevices with the customer network that may be geographically dispersed.

In the particular example shown, there are one or more gigabit Ethernet(GigE) connections 40 between the network 38 implementing the VPNs andextranet hubs 28 at respective data centers for the cloud providers 12.The extranet hubs 28 may include a physical port for the GigEconnection. To provide privacy over a shared physical connection 40,each shared connection may be provisioned with a customer specific VLANbetween the customer and a cloud provider. So, separate and dedicatedVLANs are established to provide a virtual or logical connection betweenthe customers and the cloud provider over the respective VPNs. Further,each VLAN has its own particular identification. Still referring to FIG.3, to provision the private interconnections, the system includes allowstatements or other route target mechanisms that can be utilized byvirtual routing and forwarding (VRF) tables to provision the network.The allow statements in this example are logical descriptions of how theVPNs are provisioned to provide the described connections betweencustomers and providers. For example, Allow 1:100>CP3, Allow 1:4>E1indicates that there is a VLAN provisioned between enterprise E1 (withID 1:100) and cloud provider CP3 (with ID 1:4). Stated differently incolloquial terms, enterprise E1 can “see” cloud provider CP3, and viceversa, cloud provider CP3 can “see” enterprise E1. In contrast, forexample, none of the cloud providers can see enterprise E2, andenterprise E2 cannot see any of the cloud providers.

The system illustrated in FIG. 3 involves a layer 3 interconnectionbetween the hub and the private network. Each cloud provider has atransactional VRF for each enterprise that provides VLAN separation on acustomer basis at the hub site. Thus, cloud providers cannot see eachother, and customers may privately access the cloud provider location.Through such VLAN separation, a single port or connection may be orderedfor a hub and multiple customers may privately use the port.

With respect to QoS, as introduced above, an enterprise is able toestablish the QoS for the various cloud providers the enterpriseaccesses by way of the various systems described herein. Through theportal, a customer is able to define the QoS for each provisioned cloudservice provider, and the system maintains and manages the defined QoS.The QoS is inherited on the VPN and the VLAN with the customer. For thesake of illustration, say, for example that cloud provider CP1 isproviding a Voice over IP (VoIP) service at a first QoS priority to theenterprise E1 and cloud provider CP2 is providing some other service toenterprise E2 at a second QoS priority, lower than the first. Since inone possible implementation, only one physical link is provided betweenthe VPN and the cloud provider, with such QoS provisioning, VoIP trafficover that link may be prioritized over the other traffic, which resultsin the enterprise customer's VoIP traffic being prioritized ahead ofother traffic. In a conventional public network model, prioritization oftraffic is difficult if not impossible, and hence a customer cannotprioritize and ensure that its network phone traffic is prioritized.Similarly, in a conventional private cloud environment where separatephysical connections between an enterprise and each cloud provider arerequired, it is possible to achieve a high QoS level for a cloud servicebut at the cost of establishing and maintaining a dedicated physicalconnection to the cloud provider.

FIG. 4 is a system diagram illustrating another possible implementationaccording to aspects of the present disclosure, and particularlyillustrating the importation and exportation of route targets into VRFtables at various routers to provision the private network for theshared and private virtualizations discussed herein. This examplefurther illustrates provisioning the system so that multiple offices ofa customer may each access a cloud service. Stated differently, acustomer network with several different geographical locations withinthe network has access to cloud services through the described system.In another example and in contrast, described in more detail herein, acustomer may limit access to the system through a discrete office,device or other point of access between the larger customer network andthe cloud providers such that remote offices access the cloud providersthrough the provisioned device when authorized.

Still referring to FIG. 4, this example illustrates the provisioning ofoffices S1-S5 of enterprise E1 (identification 1:100) to access cloudservices provided by CP1 (identification 1:4). The example set outherein may be extended to any number of enterprises and cloud providers.A VLAN 34 is provisioned in the network 38 between the enterprise E1 andthe cloud provider CP1. While other enterprises are not shown in FIG. 4,such enterprises can access the cloud provider CP1 over the sharedconnection. The shared connection 32 connects the cloud provider withthe service provider network 38. In this case, the service providernetwork includes a router 42, such as a provider edge router (PE1),interconnected with the hub 28 through the shared GigE connection 32.The service provider network also includes additional routers 44, whichmay also be provider edge routers, at the intersections between theservice provider network 38 and the enterprise offices S1-S5.

Referring more particularly to the enterprise E1, a plurality of routers44 may be positioned at various offices S1-S4. In this example, fiveoffices are shown having direct access to the cloud provider CP1 by wayof the described system; any number of offices, however, is possible.Each of the routers 44 is configured with a virtual routing andforwarding (VRF) table or other form of routing table.

Generally, the relevant customer routes and the relevant cloud providerroute are exchanged with the routers in the private network that canestablish the VPN and VLAN between cloud provider and the enterprise.More particularly, the relevant cloud routes are exported to the VRF ofrouter 42 that is connected with the extranet hub 28. Similarly, therelevant enterprise routes are exported to the VRFs of routers 44connected with the customer offices. The cloud provider routes are alsoimported into the customer's virtual routing and forwarding tableincluding into the VRFs of the routers 44 servicing the various offices(e.g., S1-S5) of the customer. Conversely, the customer routes,including those routes of the office routers, are imported into thevirtual routing and forwarding tables of the router 42 servicing theextranet hub 28 or other routing devices of the cloud. Thus, the variouscustomer offices S1-S5, as well as possibly others, may access the cloudprovider through the established VPN and the shared line

Still referring to FIG. 4, the route targets for the cloud serviced byextranet hub 1:4 are exported into the routing table for PE router 42.For example, a particular cloud provider may operate numerous servers toprovide its cloud services. A range of IP addresses may be defined forreach server, and these IP addresses are the route targets exported toPE router 42. Further, the route targets for enterprise E1 are exportedinto PE routers 44. More specifically, the route targets for enterpriseE1 (1:100) are imported into PE router 38 from router 44, and the routetargets for extranet hub 1:4 are imported into PE router 44.Accordingly, enterprise E1, including each of the offices S1-S5, cancommunicate with the cloud provider CP1 serviced by extranet hub 28.Because the customer VRF table is common to all offices S1-S4, eachoffice may access could provider CP1. Additionally, customer routes1:100 for each office are also imported into the customer VRF allowingthe enterprise network to be a full mesh such that S1 may communicatewith S2, S1 may also communicate with S3 and so on.

Moreover, information packets being exchanged between the cloud providerand enterprise are tagged with the respective identifications. With thedescribed tagging and VRF provisioning, information exchanged betweenthe enterprise and the cloud provider is segregated across the privatebut shared network as well as over the shared connection with the cloudprovider.

The system illustrated in FIG. 4 also illustrates a redundant pathbetween the cloud provider CP1 and the enterprise E1. Here, the cloudprovider includes a conventional connection with the public Internet 50.The enterprise may include a public Internet facing router 52 configuredto transmit and receive (transceive) communications between the cloudprovider and the public Internet through a firewall 54. The enterprisesystem, including the various offices S1-S5, communicates with the cloudprovider through the redundant path. In this example, only office S5 isshown having a connection with the redundant path and such animplementation the other offices can redundantly communicate throughoffice S5. Alternatively, the enterprise may implement any number offirewalls and provide distinct redundant connections for the variousoffices.

FIG. 5 is a system diagram illustrating another possible implementationaccording to aspects of the present disclosure. This example illustratesprovisioning the system so that one customer location (e.g. S5) mayaccess the cloud service directly through a VPN 34 and a sharedconnection 32, and that customer location distributes routing to variousoffices (e.g. S1-54) within a customer network. In this example, thecustomer network includes offices S1-S5 with each of the officesincluding various types of customer premise equipment (CPE) includingrouters. More particularly, a router 60 at location S5 is the portalthrough which various offices S1-S4 may access the cloud serviceprovider CP1. In this example, a customer may control access to thecloud provider by offices S1-S4 and those offices cannot directly reachthe cloud provider through the VPN and shared physical connection.

In this example, the cloud provider are exported into the PE router 42proximate the hub, and imported into the virtual routing and forwardingtable of the router 44(S5) connecting the customer site S5 with theprivate network 38. Conversely, the customer route for office S5 isexported to the router 44(S5) and imported into the extranet hub router42. Here, unlike the example of FIG. 4, the customer route targets foronly location S5 are exchanged with the cloud provider routes.Conventional enterprise networks may have a central office having asingle point of access to outside resources, while other offices, whichmay be geographically remote, are connected to the central office andaccess any outside resources through the central office. The remainingoffices may be interconnected in a full mesh or otherwise. For a fullmesh, the customer 1:100 routes are exchanged between the PE routers 44associated with the various offices such that each office maycommunicate with any other office.

FIG. 6 is a system diagram illustrating another possible implementationaccording to aspects of the present disclosure. This implementation maybe considered as having a layer 2 connection with the cloud provider incontrast to the implementation illustrated in FIGS. 4 and 5 that may beconsidered as having a layer 3 connection with the cloud provider. Thisimplementation may be used when the customer employs an Ethernet networkor otherwise assumes Ethernet delivery. Thus, where the above examplesprovided logical separation of customers accessing and using the VPNthrough layer 3 route targets (policies) implemented in route tables inthe various routers of the private network, the example of FIG. 6provides traffic separation and privacy through a layer 2 pseudo wirebetween the customer routers and the cloud provider. In terms ofperformance, this example is similar to FIG. 5 in that various officescommunicate with the cloud provider though a discrete access point tothe customer network.

More particularly, a one-to-one port relationship is established betweencustomer premise equipment (e.g. a router) 62 at the enterprise and ahub 28 at the cloud provider location using a pseudo-wireimplementation. In one example, an 801.1q VLAN is established between aPE router 64 and the enterprise 62. The customer premise equipment mayinclude a firewall 66. Here, cloud provider routes are exported to therouter 62 and further imported into only one, in one example, customerVRF. Other offices, besides that office serviced by CPE 62, access thecloud provider and intercommunicate similarly to as discussed relativeto FIG. 5.

FIG. 7 describes a system illustrating another possible implementationincluding network address translation (NAT) services. Typically, anenterprise will use private internet protocol (IP) space such thatinternal IP addresses of equipment within the enterprise network mayoverlap with IP addresses in the public Internet as well as othernetworks. Stated differently, equipment within an enterprise network mayhave the same IP address as equipment outside the network, such as othercustomers using the system to access the same cloud provider, andwithout some form of translation data being routed to and from thatoverlapping address there is not a way to properly send the data.Moreover, connecting to a cloud provider, which is normally accessiblethrough the public Internet, presents security and privacy concerns.Similarly, because cloud providers are often configured to communicateover the public Internet, interacting with the private IP space of anenterprise presents challenges to the cloud provider.

In the present example, the system manages IP address overlaps andtranslation between public IP space associated with the cloud providerand private IP space of the enterprises through implementation of one ormore NAT services within the system interconnecting the cloudprovider(s) and the enterprise(s). This implementation is similar tothat shown in FIG. 4 with the addition of a NAT service at the provideredge router in communication with the cloud provider hub. In thisexample, an edge router with NAT services is in communication with thecloud provider's extranet hub.

More particularly, a VLAN 34 is provisioned in the network 38 betweenthe enterprise E1 and the cloud provider CP1. While other enterprisesare not shown in FIG. 7, such enterprises can access the cloud providerCP1 over a shared connection 32. The shared connection 32 connects thecloud provider with the service provider network 38. In this case, theservice provider network includes a router 68, such as a provider edgerouter (PE1), interconnected with the hub 28 through the sharedconnection 32, which may be a GigE connection. Unlike router 42 of FIG.4, the router 68 also include NAT services. Generally speaking, thecloud provider side of the router 68 involves public IP space whereasthe customer side of the router 68 involves the private IP space of thecustomer. The Router/NAT 68 advertises a public IP space to the cloudprovider, which may be geographically unique, and then translates thepackets received to the appropriate addresses of the private customerspace. The service provider network also includes additional routers 44,which may also be provider edge routers, at the intersections betweenthe service provider network 38 and the enterprise offices S1-S5. Theserouters do not include NAT services, in one example. However, NATservices may be provided in other locations of the private network, andNAT services may be provided in some or all of the routers servicingother hubs of CP1 as well as hubs of other cloud providers.

Referring more particularly to the enterprise E1, a plurality of routers44 may be positioned at various offices S1-S5. In this example, fiveoffices are shown having direct access to the cloud provider CP1 by wayof the described system; any number of offices, however, is possible.Each of the routers 44 is configured with a virtual routing andforwarding (VRF) table or other form of routing table.

Generally, the relevant customer routes and the relevant cloud providerroutes are exchanged with the routers in the private network that canestablish the VPN and VLAN between cloud provider and the enterprise.More particularly, the relevant cloud routes are exported to the VRF ofrouter 42 that is connected with the extranet hub 28. Similarly, therelevant enterprise routes are exported to the VRFs of routers 44connected with the customer offices. The cloud provider routes are alsoimported into the customer's virtual routing and forwarding tableincluding into the VRFs of the routers 44 servicing the various offices(e.g., S1-S5) of the customer. Conversely, the customer routes,including those routes of the office routers, are imported into thevirtual routing and forwarding tables of the router 42 servicing theextranet hub 28 or other routing devices of the cloud. Thus, the variouscustomer offices S1-S5, as well as possibly others, may access the cloudprovider through the established VPN and the shared line 32.

Still referring to FIG. 7, the route targets for the cloud serviced byextranet hub 1:4 are exported into the routing table for PE router 42.For example, a particular cloud provider may operate numerous servers toprovide its cloud services. A range of IP addresses may be defined foreach server, and these IP addresses are the route targets exported to PErouter 42. Further, the route targets for enterprise E1 are exportedinto PE routers 44. More specifically, the route targets for enterpriseE1 (1:100) are imported into PE router 38 from router 44, and the routetargets for extranet hub 1:4 are imported into PE router 44. The NATprovides translation between the customer route targets and the cloudprovider route targets. Accordingly, enterprise E1, including each ofthe offices S1-S5, can communicate with the cloud provider CP1 servicedby extranet hub 28. Because the customer VRF table is common to alloffices S1-S5, each office may access could provider CP1. Additionally,customer routes 1:100 for each office are also imported into thecustomer VRF allowing the enterprise network to be a full mesh such thatS1 may communicate with S2, S1 may also communicate with S3 and so on.

Moreover, information packets being exchanged between the cloud providerand enterprise are tagged with the respective identifications. With thedescribed tagging, VRF provisioning, and NAT service(s) informationexchanged between the enterprise and the cloud provider is segregatedacross the private but shared network as well as over the sharedconnection with the cloud provider.

In other possible examples, edge routers with NAT services may bepositioned in proximity to and in communication with extranet hubs atvarious different cloud provider locations. Further, the cloud providermay provide additional and/or redundant services at discrete locations.In such an implementation, the discrete locations may include there ownidentifications. So, for example if CP1 has three data centers providingsome or all of its various services, the hubs for those data centers mayhave identifications 1:4, 1:5 and 1:6. However, the hubs export the sameIP address or some portion of the IP address, e.g., 1.1.1.1., to therouters for some associated service. However, each router includesinformation concerning the hub to which it is attached which includesgeographic information or some indication of its logical location in thenetwork. Thus, when a customer attempts to access the cloud service, theprivate network is able to route the request to the nearest router 68that provides access to the cloud servicing the request. As each PErouter includes the associated specific cloud routes, the router thatservices the request can privately communicate with the associated cloudresources that will service the request.

FIG. 8 illustrates one particular way to implement the system describedin FIG. 7. Here, a GigE/EVPL backhaul connection 80 is provisionedbetween a cloud provider (e.g., CP2) and a provider edge router 68including NAT services within the network 28 providing VPN services. Thenetwork, which may include additional PE routers as well as otherrouting equipment including a backbone network, is connected with anenterprise (e.g., E1) over a physical connection that can be shared withother enterprises. Thus, the PE router and NAT are logically positionedbetween the enterprise and the cloud provider. This implementation ishelpful for cloud providers that conventionally only support publicinterfaces and network addresses.

FIG. 9 is a diagram illustrating another system illustrating a way toprovide private virtualized cloud services to a plurality of customersas well as providing NAT services.

In this example implementation, a cloud provider CP1 is shown having twodiscrete data centers with associated hubs having identifications 1:4and 1:5, respectively. These data centers or cloud provider locationsmay be geographically separated in different locations, say Seattle andLondon. The system, besides allowing a plurality of customers E1-EN toaccess each cloud provider location over a shared connection(s) 32 tothe particular location, also can route customer communications to thenearest (geographically and/or logically) cloud provider data centerproviding the requested service. Thus, a customer in Europe requesting acloud service might be routed to the cloud provider location in Londonwhereas a customer in California might be routed to the cloud providerlocation in Seattle. Should either location go off-line for any reason,communications are automatically routed to the other on-line location.

In this implementation, one or more first PE routers 90 are shown inproximity to the cloud provider hubs, and a second PE router 92implementing a NAT service with virtual domains somewhere in the privatenetwork 28 between the cloud provider and the plurality of customersconfigured to communicate with the cloud provider. Alternatively, the PErouter at the hub or other routers may also include the NAT service. Apeering connection using border gateway protocol session (BGP) isestablished between the first PE router(s) at the cloud provider hub(s)and the second PE router implementing the NAT service in the network.The BGP session, in one implementation, is created only once and henceminimizes the cloud provider's overhead. Moreover, customers may sharethe BGP session and may be provisioned in the system to access the cloudprovider or decommissioned so access is not allowed without the cloudprovider being involved in the changes (i.e., customer additions anddeletions are transparent to the cloud provider).

In this example, four customers E1 -E4 are illustrated as using thesystem to obtain cloud services. The NAT includes four correspondingvirtual domains. A virtual domain implements policies, quality ofservices and other customer (enterprise) unique routing parameters. Inone example, each VDOM defines a virtualized firewall instance for eachcustomer (enterprise) where each virtualized firewall implementspolicies specific to each customer. In this example, the NATimplementing the VDOM may be implemented in a single router wheremultiple customers may be provisioned to access the BGP session and theassociated cloud provider hub. In this example, a third set of PErouters 94 are associated with each of the four customers C1-C4.

As with other implementations discussed herein, various route targets ofthe customer and the cloud provider are imported and exported into VRF'sof the PE routers. The route targets for the hub(s) are exported to thefirst PE router(s), while the customer routes are exported to the thirdPE router(s). Additionally, the cloud provider routes are imported intothe second PE router implementing the NAT and similarly the enterpriseroutes are imported in the second PE router. In contrast to otherimplementations, the cloud and enterprise route targets are importedinto the VDOMs.

FIG. 10 is a system diagram further illustrating the implementation setout in FIG. 9, and particularly illustrating the association of customerand cloud VRF routing in the VDOM established in the PE/NAT 92. The NATmay implement 1:1 translation or 1:many translation.

FIG. 11 is a diagram illustrating class of service or QOS support withinthe described systems. In this system, the network provider can managethe customer's indicated class of service. So, for example, if anenterprise is obtaining two types of services from one or more cloudproviders, the enterprise can prioritize or otherwise define a class ofservice for the two services. In this example, the customer may define,such as through the portal, a first high priority to email and a secondintermediate priority to voice. The system provides a signature for thedifferent types of services provided. So, for example, the cloudprovider may provide a range of IP addresses associated with email andsecond range of IP addresses associated with voice services. The systemthen can identify packets from the IP addresses and make routingdecisions according to the defined class of service. In anotheralternative, the cloud provider may include port information fordifferent types of services, and the system may append unique packetheader information depending on the port from which the packet isreceived. The system may then make class of service routing decisionsbased on the appended packet header information.

A class of service module 96 may be provisioned in the PE router 38 atthe cloud hub. The module 96 may be provisioned with a table mapping thevarious services to the associated class of service. Thus, for example,the table would include a map between email packets and a high class ofservice and voice packets and intermediate class of service. Whenpackets are received, the module performs a table look-up to extract theclass of service, and makes subsequent routing decisions based on theclass of service. So, for example, if network bandwidth between thecustomer and cloud provider has been substantially consumed, the packetswith high priority will be processed and transmitted before thosepackets, which can be queued, are processed and transmitted.

The system can manage or implement various class of servicearrangements, including 4q (four different classes of services) and 6q(six different classes of service).

FIG. 12 is a schematic diagram of a computer system 100 upon whichembodiments discussed herein may be carried out and implemented.

According to the present example, the computer system 100 includes a bus101 (i.e., interconnect), at least one processor 102, at least onecommunications port 103, a main memory 104, a removable storage media105, a read-only memory 106, and a mass storage 107. Processor(s) 102can be any known processor, such as, but not limited to, an Inte^(l)®Itaniu^(m)® or Itanium²® processor(s), AM^(D)® Optero^(n)® or AthlonM^(P)® processor(s), or Motorol^(a)® lines of processors. Communicationsports 103 can be any of an RS-232 port for use with a modem baseddial-up connection, a 10/100 Ethernet port, a Gigabit port using copperor fiber, or a USB port. Communications port(s) 103 may be chosendepending on a network such as a Local Area Network (LAN), a Wide AreaNetwork (WAN), service provider network, customer network, cloud serviceprovider network, or any network to which the computer system 100connects (e.g., service provider network 190). The computer system 100may be in communication with peripheral devices (e.g., display screen130, input device 116) via Input/Output (I/O) port 109.

Main memory 104 can be Random Access Memory (RAM), or any other dynamicstorage device(s) commonly known in the art. Read-only memory 106 can beany static storage device(s) such as Programmable Read-Only Memory(PROM) chips for storing static information such as instructions forprocessor 102. Mass storage 107 can be used to store information andinstructions. For example, hard disks such as the Adaptec® family ofSmall Computer Serial Interface (SCSI) drives, an optical disc, an arrayof disks such as Redundant Array of Independent Disks (RAID), such asthe Adaptec® family of RAID drives, or any other mass storage devicesmay be used.

Bus 101 communicatively couples processor(s) 102 with the other memory,storage and communications blocks. Bus 101 can be a PCI/PCI-X, SCSI, orUniversal Serial Bus (USB) based system bus (or other) depending on thestorage devices used. Removable storage media 105 can be any kind ofexternal hard-drives, floppy drives, IOMEGA® Zip Drives, CompactDisc-Read Only Memory (CD-ROM), Compact Disc-Re-Writable (CD-RW),Digital Video Disk-Read Only Memory (DVD-ROM), etc.

Embodiments herein may be provided as a computer program product, whichmay include a machine-readable medium having stored thereoninstructions, which may be used to program a computer (or otherelectronic devices) to perform a process. The machine-readable mediummay include, but is not limited to, floppy diskettes, optical discs,CD-ROMs, magneto-optical disks, ROMs, RAMs, erasable programmableread-only memories (EPROMs), electrically erasable programmableread-only memories (EEPROMs), magnetic or optical cards, flash memory,or other type of media/machine-readable medium suitable for storingelectronic instructions. Moreover, embodiments herein may also bedownloaded as a computer program product, wherein the program may betransferred from a remote computer to a requesting computer by way ofdata signals embodied in a carrier wave or other propagation medium viaa communication link (e.g., modem or network connection).

As shown, main memory 104 is encoded with cloud network application150-1 that supports functionality as discussed above and as discussedfurther below. Cloud network application 150-1 (and/or other resourcesas described herein) can be embodied as software code such as dataand/or logic instructions (e.g., code stored in the memory or on anothercomputer readable medium such as a disk) that supports processingfunctionality according to different embodiments described herein.During operation of one embodiment, processor(s) 102 accesses mainmemory 104 via the use of bus 101 in order to launch, run, execute,interpret or otherwise perform the logic instructions of the cloudnetwork application 150-1. Execution of cloud network application 150-1produces processing functionality in cloud network process 150-2. Inother words, the cloud network process 150-2 represents one or moreportions of the cloud network application 150-1 performing within orupon the processor(s) 102 in the computer system 100.

It should be noted that, in addition to the cloud network process 150-2that carries out method operations as discussed herein, otherembodiments herein include the cloud network application 150-1 itself(i.e., the un-executed or non-performing logic instructions and/ordata). The cloud network application 150-1 may be stored on a computerreadable medium (e.g., a repository) such as a floppy disk, hard disk orin an optical medium. According to other embodiments, the cloud networkapplication 150-1 can also be stored in a memory type system such as infirmware, read only memory (ROM), or, as in this example, as executablecode within the main memory 104 (e.g., within Random Access Memory orRAM). For example, cloud network application 150-1 may also be stored inremovable storage media 105, read-only memory 106, and/or mass storagedevice 107.

As discussed herein, embodiments discussed herein include various stepsor operations. A variety of these steps may be performed by hardwarecomponents or may be embodied in machine-executable instructions, whichmay be used to cause a general-purpose or special-purpose processorprogrammed with the instructions to perform the operations.Alternatively, the steps may be performed by a combination of hardware,software, and/or firmware.

FIG. 13 includes a flowchart according to embodiments herein. Therectangular elements are herein denoted as “steps” and representcomputer software instructions or groups of instructions that carry outsuch functions. The flow diagrams do not necessarily depict the syntaxof any particular programming language. Rather, the flow diagramsillustrate the functional information one of ordinary skill in the artcould use to fabricate circuits or to generate computer software (or ahybrid of both circuits and software code) to carry out the features asdescribed herein.

It should be noted that many routine program elements, such asinitialization of loops and variables and the use of temporary variablesare inherent in the flowcharts. It will be appreciated by those ofordinary skill in the art that unless otherwise indicated herein, theparticular sequence of steps described is illustrative only and can bevaried without departing from the spirit of the invention. Thus, unlessotherwise stated the steps described below are unordered meaning that,when possible, the steps can be performed in any convenient or desirableorder.

Now, more specifically, FIG. 13 shows a flow chart 200 of processingsteps representing processing operations performed by the cloud networkprocess 150 (i.e., cloud network application 150-1 and/or the run-timeimplementation of cloud network process 150-2) in accordance with oneexample embodiment.

In step 205, the cloud network process 150 provides virtual privatenetwork (VPN) access across a service provider network for each of aplurality of customers of the service provider network.

In step 210, the cloud network process 150 provides private accessacross the service provider network between at least one of theplurality of customers and at least one of a plurality of cloud serviceproviders associated with the service provider network.

In step 215, the cloud network process 150 implements Virtual Forwardingand Routing (VRF) to enable a secure multi-tenancy environment in theservice provider network among the plurality of customers and theplurality of cloud service providers.

In step 220, the cloud network process 150 implements Network AddressTranslation (NAT) to enable scaling of private access between theplurality of customers and the plurality of cloud service providersacross the service provider network.

In step 225, the cloud network process 150 causes an edge router in theservice provider network to implement NAT between at least one of theplurality of customers and at least one of the plurality of cloudservice providers.

Although the present invention has been described with reference tovarious embodiments, it will be understood that the invention is notlimited to the details thereof. Various modifications and substitutionswill occur to those of ordinary skill in the art. All such substitutionsare intended to be embraced within the scope of the invention as definedin the appended claims.

What is claimed is:
 1. A computer-implemented method for providingvirtualized cloud services across a service provider network, thecomputer-implemented method comprising: providing virtual privatenetwork (VPN) access across the service provider network for each of aplurality of customers of the service provider network; providingprivate access across the service provider network between at least oneof the plurality of customers and at least one of a plurality of cloudservice providers associated with the service provider network, theprivate access being over a shared physical connection between theservice provider network and the at least one of the plurality of cloudproviders; implementing Virtual Forwarding and Routing (VRF) to enable asecure multi-tenancy environment in the service provider network amongthe plurality of customers and the plurality of cloud service providers;and implementing Network Address Translation (NAT) to enable scaling ofprivate access between the plurality of customers and the plurality ofcloud service providers across the service provider network.
 2. Thecomputer-implemented method of claim 1 further comprising: exporting aplurality of route targets from a cloud provider network configured toprovide one or more services of the cloud provider to a first VRF of atleast one first router of the service provider network; exporting aplurality of route targets from a customer network of at least one ofthe plurality of customers to a second VRF of at least one second routerof the service provider network; importing the plurality of routetargets from the first VRF at the second VRF; and importing theplurality of route targets from the second VRF at the first VRF.
 3. Thecomputer implemented method of claim 2 wherein the at least one secondrouter comprises a plurality of provider edge routers for acorresponding plurality of customer offices of the customer network,each of the plurality of routers including a VRF with the cloud providernetwork route targets such that the offices have secure private accessto at least one service of the cloud provider.
 4. Thecomputer-implemented method as recited in claim 1, further comprisingcausing an edge router in the service provider network to implement NATbetween at least one of the plurality of customers and at least one ofthe plurality of cloud service providers.
 5. The computer-implementedmethod as recited in claim 4 further comprising: establishing a bordergateway protocol (BGP) session between the edge router in the serviceprovider network and a second router of the service provider network,the second router having a physical connection with the cloud providernetwork; and providing private shared access between each customernetwork and the cloud provider network using the BGP session andphysical connection with the cloud provider network.
 6. The computerimplemented method of claim 5 further comprising: establishing aplurality of virtual domains unique to each customer accessing the cloudprovider network, the virtual domains at the edge router implementingthe NAT of the hub of the cloud provider network.
 7. Thecomputer-implemented method as recited in claim 1, further comprising:tagging a first unique identification to data packets being routed froma cloud provider network to a particular customer through a particularVPN of the service provider network; tagging a second uniqueidentification to data packets being routed from the particular customerthrough the particular VPN of the service provider network to the cloudprovider network; whereby the data packets being routed from the cloudprovider network to the particular customer network, and the datapackets being routed form the particular customer network and the cloudprovider network are private between the particular customer network andthe particular cloud provider network.
 8. The computer implementedmethod as recited in claim 1 further comprising implementing class ofservice routing among packets distributed across the VPN between aparticular cloud provider and a particular customer.